Evaluation of elicitation methods to quantify Bayes linear models

The Bayes linear methodology allows decision makers to express their subjective beliefs and adjust these beliefs as observations are made. It is similar in spirit to probabilistic Bayesian approaches, but differs as it uses expectation as its primitive. While substantial work has been carried out in Bayes linear analysis, both in terms of theory development and application, there is little published material on the elicitation of structured expert judgement to quantify models. This paper investigates different methods that could be used by analysts when creating an elicitation process. The theoretical underpinnings of the elicitation methods developed are explored and an evaluation of their use is presented. This work was motivated by, and is a precursor to, an industrial application of Bayes linear modelling of the reliability of defence systems. An illustrative example demonstrates how the methods can be used in practice

Precalibrating an intermediate complexity climate model

Credible climate predictions require a rational quantification of uncertainty, but full Bayesian calibration requires detailed estimates of prior probability distributions and covariances, which are difficult to obtain in practice. We describe a simplified procedure, termed precalibration, which provides an approximate quantification of uncertainty in climate prediction, and requires only that uncontroversially implausible values of certain inputs and outputs are identified. The method is applied to intermediate-complexity model simulations of the Atlantic meridional overturning circulation (AMOC) and confirms the existence of a cliff-edge catastrophe in freshwaterforcing input space. When uncertainty in 14 further parameters is taken into account, an implausible, AMOC-off, region remains as a robust feature of the model dynamics, but its location is found to depend strongly on values of the other parameters

EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

This is the final version of the article. Available from BioMed Central via the DOI in this recordBACKGROUND: Expert opinion is often sought to complement available information needed to inform model-based economic evaluations in health technology assessments. In this context, we define expert elicitation as the process of encoding expert opinion on a quantity of interest, together with associated uncertainty, as a probability distribution. When availability for face-to-face expert elicitation with a facilitator is limited, elicitation can be conducted remotely, overcoming challenges of finding an appropriate time to meet the expert and allowing access to experts situated too far away for practical face-to-face sessions. However, distance elicitation is associated with reduced response rates and limited assistance for the expert during the elicitation session. The aim of this study was to inform the development of a remote elicitation tool by exploring the influence of mode of elicitation on elicited beliefs. METHODS: An Excel-based tool (EXPLICIT) was developed to assist the elicitation session, including the preparation of the expert and recording of their responses. General practitioners (GPs) were invited to provide expert opinion about population alcohol consumption behaviours. They were randomised to complete the elicitation by either a face-to-face meeting or email. EXPLICIT was used in the elicitation sessions for both arms. RESULTS: Fifteen GPs completed the elicitation session. Those conducted by email were longer than the face-to-face sessions (13 min 30 s vs 10 min 26 s, p = 0.1) and the email-elicited estimates contained less uncertainty. However, the resulting aggregated distributions were comparable. CONCLUSIONS: EXPLICIT was useful in both facilitating the elicitation task and in obtaining expert opinion from experts via email. The findings support the opinion that remote, self-administered elicitation is a viable approach within the constraints of HTA to inform policy making, although poor response rates may be observed and additional time for individual sessions may be required.This paper presents independent research funded by the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for the South West Peninsula

Using modeling tools for implementing feasible land use and nature conservation governance systems in small islands e The Pico Island (Azores) case-study

The present study deals with the development of systematic conservation planning as management instrument in small oceanic islands, ensuring open systems of governance, and able to integrate an informed and involved participation of the stakeholders. Marxan software was used to define management areas according a set of alternative land use scenarios considering different conservation and management paradigms. Modeled conservation zones were interpreted and compared with the existing protected areas allowing more fused information for future trade-outs and stakeholder's involvement. The results, allowing the identification of Target Management Units (TMU) based on the consideration of different development scenarios proved to be consistent with a feasible development of evaluation approaches able to support sound governance systems. Moreover, the detailed geographic identification of TMU seems to be able to support participated policies towards a more sustainable management of the entire islan

A comparison of two methods for expert elicitation in health technology assessments.

BACKGROUND: When data needed to inform parameters in decision models are lacking, formal elicitation of expert judgement can be used to characterise parameter uncertainty. Although numerous methods for eliciting expert opinion as probability distributions exist, there is little research to suggest whether one method is more useful than any other method. This study had three objectives: (i) to obtain subjective probability distributions characterising parameter uncertainty in the context of a health technology assessment; (ii) to compare two elicitation methods by eliciting the same parameters in different ways; (iii) to collect subjective preferences of the experts for the different elicitation methods used. METHODS: Twenty-seven clinical experts were invited to participate in an elicitation exercise to inform a published model-based cost-effectiveness analysis of alternative treatments for prostate cancer. Participants were individually asked to express their judgements as probability distributions using two different methods - the histogram and hybrid elicitation methods - presented in a random order. Individual distributions were mathematically aggregated across experts with and without weighting. The resulting combined distributions were used in the probabilistic analysis of the decision model and mean incremental cost-effectiveness ratios and the expected values of perfect information (EVPI) were calculated for each method, and compared with the original cost-effectiveness analysis. Scores on the ease of use of the two methods and the extent to which the probability distributions obtained from each method accurately reflected the expert's opinion were also recorded. RESULTS: Six experts completed the task. Mean ICERs from the probabilistic analysis ranged between £162,600-£175,500 per quality-adjusted life year (QALY) depending on the elicitation and weighting methods used. Compared to having no information, use of expert opinion decreased decision uncertainty: the EVPI value at the £30,000 per QALY threshold decreased by 74-86 % from the original cost-effectiveness analysis. Experts indicated that the histogram method was easier to use, but attributed a perception of more accuracy to the hybrid method. CONCLUSIONS: Inclusion of expert elicitation can decrease decision uncertainty. Here, choice of method did not affect the overall cost-effectiveness conclusions, but researchers intending to use expert elicitation need to be aware of the impact different methods could have.This paper presents independent research funded by the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for the South West Peninsula

Index-based approach for estimating vulnerability of Arctic biota to oil spills

Risk of an Arctic oil spill has become a global matter of concern. Climate change induced opening of shipping routes increases the Arctic maritime traffic which exposes the area to negative impacts of potential maritime accidents. Still, quantitative analyses of the likely environmental impacts of such accidents are scarce, and our understanding of the uncertainties related to both accidents and their consequences is poor. There is an obvious need for analysis tools that allow us to systematically analyze the impacts of oil spills on Arctic species, so the risks can be taken into account when new sea routes or previously unexploited oil reserves are utilized. In this paper, an index‐based approach is developed to study exposure potential (described via probability of becoming exposed to spilled oil) and sensitivity (described via oil‐induced mortality and recovery) of Arctic biota in the face of an oil spill. First, a conceptual model presenting the relevant variables that contribute to exposure potential and sensitivity of key Arctic marine functional groups was built. Second, based on an extensive literature review, a probabilistic estimate was assigned for each variable, and the variables were combined to an index representing the overall vulnerability of Arctic biota. The resulting index can be used to compare the relative risk between functional groups and accident scenarios. Results indicate that birds have the highest vulnerability to spilled oil, and seals and whales the lowest. Polar bears’ vulnerability varies greatly between seasons, while ice seals’ vulnerability remains the same in every accident scenario. Exposure potential of most groups depends strongly on type of oil, whereas their sensitivity contains less variation.Peer reviewe

Exploring Antarctic subglacial lakes with scientific probes: a formal probabilistic approach for operational risk management

Since their discovery, Antarctic subglacial lakes have become of great interest to the science community. It is hypothesized that they may hold unique forms of biological life and that they hold detailed sedimentary records of past climate change. According to the latest inventory, a total of 387 subglacial lakes have been identified in Antarctica (Wright and Siegert, 2011). However, exploration using scientific probes has yet to be performed. We propose a generic, formal approach to manage the operational risk of deploying probes during clean access to subglacial lake exploration. A representation of the entire probe deployment process is captured in a Markov chain. The transition from one state to the next depends on several factors, including reliability of components and processes. We use fault trees to quantify the probability of failure of the complex processes that must take place to facilitate the transition from one state to another. Therefore, the formal framework consists of integrating a Markov chain, fault trees, component and subsystem reliability data and expert judgment. To illustrate its application we describe how the approach can be used to address a series of what-if scenarios, using the intended Ellsworth Subglacial Lake probe deployment as a case study